Replace init secrets init container with helm #5557

ddl-ebrown · 2024-07-12T04:02:18Z

Tracking issue

Why are the changes needed?

Removes deployment complexity

A couple of other things to call out here:

It may be useful to deprecate / remove the code previously used to generate secrets if it's not otherwise needed (not sure what the standalone binary chart does for init)
The ClusterRole should be updated to remove the ability to write / delete secrets - as it stands, https://github.com/flyteorg/flyte/blob/master/charts/flyte-core/values.yaml#L71-L94 is extremely permissive and a bit of a security liability (why does flyteadmin need access to all secrets in the cluster?). Using Helm to init the secret allows for restricting the service account more appropriately

What changes were proposed in this pull request?

Remove core flyteadmin init-secret initContainer

A separate container instance that requires Kubernetes API access is
unnecessary when Helm is able to generate the same secret values in
the client before submitting resources to Kubernetes.

The declarative approach here is identical to what the code does in
https://github.com/flyteorg/flyte/blob/master/flyteadmin/auth/init_secrets.go#L80-L151
The lookup function is used so that on upgrades, the secret is not
regenerated -- the existing values in the cluster are used.
Note that Helm always creates secret resources before deployment
resources, so the secret values are guaranteed to be available before
flyteadmin starts

Formalize oidc_client_secret in flyte-admin-secrets

When setting up integrations with an IdP like Keycloak, the
oidc_client_secret must also be set in flyte-admin-secrets.

Formalize that and make it more discoverable

How was this patch tested?

Setup process

Screenshots

Check all the applicable boxes

I updated the documentation accordingly.
All new and existing tests passed.
All commits are signed-off.

Related PRs

Docs link

codecov · 2024-07-12T04:06:17Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 60.97%. Comparing base (81afb76) to head (b3e424e).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5557      +/-   ##
==========================================
- Coverage   60.98%   60.97%   -0.02%     
==========================================
  Files         796      796              
  Lines       51647    51647              
==========================================
- Hits        31498    31492       -6     
- Misses      17249    17255       +6     
  Partials     2900     2900

Flag	Coverage Δ
unittests-datacatalog	`69.31% <ø> (ø)`
unittests-flyteadmin	`58.69% <ø> (-0.05%)`	⬇️
unittests-flytecopilot	`17.79% <ø> (ø)`
unittests-flytectl	`67.44% <ø> (ø)`
unittests-flyteidl	`79.06% <ø> (ø)`
unittests-flyteplugins	`61.85% <ø> (ø)`
unittests-flytepropeller	`57.42% <ø> (ø)`
unittests-flytestdlib	`65.68% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

ddl-ebrown · 2024-07-12T05:33:32Z

script/generate_helm.sh

@@ -56,7 +56,8 @@ ${GOPATH:-~/go}/bin/helm-docs -c ${DIR}/../charts/

 # This section is used by GitHub workflow to ensure that the generation step was run
 if [ -n "$DELTA_CHECK" ]; then
-	DIRTY=$(git status --porcelain)
+	# find only deleted or removed lines, not replaced values
+	DIRTY=$(git diff --word-diff | grep '^[{\[]')


The CI check needs to be updated to understand that some values may change when regenerating Helm charts

An alternative approach would be to add values.yaml entries and plumb them through -- but that feels a bit overkill

cc @eapolinario please take a look

- A separate container instance that requires Kubernetes API access is unnecessary when Helm is able to generate the same secret values in the client before submitting resources to Kubernetes. The declarative approach here is identical to what the code does in https://github.com/flyteorg/flyte/blob/master/flyteadmin/auth/init_secrets.go#L80-L151 - The lookup function is used so that on upgrades, the secret is not regenerated -- the existing values in the cluster are used. - Note that Helm always creates secret resources before deployment resources, so the secret values are guaranteed to be available before flyteadmin starts - Update helm chart regenerate check so that it only fails if the diff has new or removed lines Signed-off-by: ddl-ebrown <[email protected]>

- When setting up integrations with an IdP like Keycloak, the oidc_client_secret must also be set in flyte-admin-secrets. Formalize that and make it more discoverable Signed-off-by: ddl-ebrown <[email protected]>

davidmirror-ops

Also reproduced this and works as expected. Thank you!

@ddl-ebrown do you think you could PR the cleanup tasks you mention in the description? I think it makes sense:

It may be useful to deprecate / remove the code previously used to generate secrets if it's not otherwise needed (not sure what the standalone binary chart does for init)
The ClusterRole should be updated to remove the ability to write / delete secrets - as it stands, https://github.com/flyteorg/flyte/blob/master/charts/flyte-core/values.yaml#L71-L94 is extremely permissive and a bit of a security liability (why does flyteadmin need access to all secrets in the cluster?). Using Helm to init the secret allows for restricting the service account more appropriately

ddl-ebrown · 2024-07-25T20:15:09Z

Also reproduced this and works as expected. Thank you!

@ddl-ebrown do you think you could PR the cleanup tasks you mention in the description? I think it makes sense:

It may be useful to deprecate / remove the code previously used to generate secrets if it's not otherwise needed (not sure what the standalone binary chart does for init)
The ClusterRole should be updated to remove the ability to write / delete secrets - as it stands, master/charts/flyte-core/values.yaml#L71-L94 is extremely permissive and a bit of a security liability (why does flyteadmin need access to all secrets in the cluster?). Using Helm to init the secret allows for restricting the service account more appropriately

Happy to remove any dead code used to generate these values -- that should be easy.

I'd like someone on your side to confirm the ClusterRole changes though, and I think it might be best to do a separate pass on cleaning up the default ClusterRole config in another PR since the issue is a little orthogonal (my hunch is that a bunch of those perms are unnecessary for flyteadmin). What do you think?

davidmirror-ops · 2024-07-26T21:18:51Z

Also reproduced this and works as expected. Thank you!
@ddl-ebrown do you think you could PR the cleanup tasks you mention in the description? I think it makes sense:

It may be useful to deprecate / remove the code previously used to generate secrets if it's not otherwise needed (not sure what the standalone binary chart does for init)
The ClusterRole should be updated to remove the ability to write / delete secrets - as it stands, master/charts/flyte-core/values.yaml#L71-L94 is extremely permissive and a bit of a security liability (why does flyteadmin need access to all secrets in the cluster?). Using Helm to init the secret allows for restricting the service account more appropriately

Happy to remove any dead code used to generate these values -- that should be easy.

I'd like someone on your side to confirm the ClusterRole changes though, and I think it might be best to do a separate pass on cleaning up the default ClusterRole config in another PR since the issue is a little orthogonal (my hunch is that a bunch of those perms are unnecessary for flyteadmin). What do you think?

Yep, the changes to the ClusterRole should be a separate PR, as the removal of the unused code.

ddl-ebrown · 2024-07-27T17:50:34Z

Also reproduced this and works as expected. Thank you!
@ddl-ebrown do you think you could PR the cleanup tasks you mention in the description? I think it makes sense:

It may be useful to deprecate / remove the code previously used to generate secrets if it's not otherwise needed (not sure what the standalone binary chart does for init)
The ClusterRole should be updated to remove the ability to write / delete secrets - as it stands, master/charts/flyte-core/values.yaml#L71-L94 is extremely permissive and a bit of a security liability (why does flyteadmin need access to all secrets in the cluster?). Using Helm to init the secret allows for restricting the service account more appropriately

Happy to remove any dead code used to generate these values -- that should be easy.
I'd like someone on your side to confirm the ClusterRole changes though, and I think it might be best to do a separate pass on cleaning up the default ClusterRole config in another PR since the issue is a little orthogonal (my hunch is that a bunch of those perms are unnecessary for flyteadmin). What do you think?

Yep, the changes to the ClusterRole should be a separate PR, as the removal of the unused code.

Ok, so then to get this one merged we just need to get the CI check green

ddl-ebrown force-pushed the replace-init-secrets-initContainer-with-helm branch 2 times, most recently from 873cbab to fc035cc Compare July 12, 2024 05:20

ddl-ebrown commented Jul 12, 2024

View reviewed changes

ddl-ebrown force-pushed the replace-init-secrets-initContainer-with-helm branch from fc035cc to 4a95b31 Compare July 12, 2024 05:46

ddl-ebrown added 2 commits July 12, 2024 12:49

Formalize oidc_client_secret in flyte-admin-secrets

b3e424e

- When setting up integrations with an IdP like Keycloak, the oidc_client_secret must also be set in flyte-admin-secrets. Formalize that and make it more discoverable Signed-off-by: ddl-ebrown <[email protected]>

ddl-ebrown force-pushed the replace-init-secrets-initContainer-with-helm branch from 4a95b31 to b3e424e Compare July 12, 2024 20:04

ddl-ebrown requested a review from davidmirror-ops July 12, 2024 22:13

davidmirror-ops approved these changes Jul 25, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace init secrets init container with helm #5557

Replace init secrets init container with helm #5557

ddl-ebrown commented Jul 12, 2024 •

edited

Loading

codecov bot commented Jul 12, 2024 •

edited

Loading

ddl-ebrown Jul 12, 2024

davidmirror-ops Jul 29, 2024

davidmirror-ops left a comment

ddl-ebrown commented Jul 25, 2024

davidmirror-ops commented Jul 26, 2024

ddl-ebrown commented Jul 27, 2024

Replace init secrets init container with helm #5557

Are you sure you want to change the base?

Replace init secrets init container with helm #5557

Conversation

ddl-ebrown commented Jul 12, 2024 • edited Loading

Tracking issue

Why are the changes needed?

What changes were proposed in this pull request?

Remove core flyteadmin init-secret initContainer

Formalize oidc_client_secret in flyte-admin-secrets

How was this patch tested?

Setup process

Screenshots

Check all the applicable boxes

Related PRs

Docs link

codecov bot commented Jul 12, 2024 • edited Loading

Codecov Report

ddl-ebrown Jul 12, 2024

Choose a reason for hiding this comment

davidmirror-ops Jul 29, 2024

Choose a reason for hiding this comment

davidmirror-ops left a comment

Choose a reason for hiding this comment

ddl-ebrown commented Jul 25, 2024

davidmirror-ops commented Jul 26, 2024

ddl-ebrown commented Jul 27, 2024

ddl-ebrown commented Jul 12, 2024 •

edited

Loading

codecov bot commented Jul 12, 2024 •

edited

Loading